fix(recall): collapse duplicate raw facts before truncation#2577
Closed
Cybercommanders wants to merge 1 commit into
Closed
fix(recall): collapse duplicate raw facts before truncation#2577Cybercommanders wants to merge 1 commit into
Cybercommanders wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a configurable deduplication pass for internal recall results so near-identical raw world/experience facts are collapsed before truncation, improving recall diversity under tight token budgets.
Changes:
- Introduces recall dedup helpers (
normalize_recall_dedup_text,collapse_near_duplicate_raw_facts) and wires them intoMemoryEnginerecall flow. - Adds
HINDSIGHT_API_RECALL_DEDUP_THRESHOLDconfig + validation, and documents it in env templates and developer configuration docs. - Adds unit tests for dedup behavior and config wiring/defaults.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| hindsight-embed/hindsight_embed/env.example | Documents the new recall dedup env var in the embed template. |
| hindsight-docs/docs/developer/configuration.md | Adds the new HINDSIGHT_API_RECALL_DEDUP_THRESHOLD entry to configuration reference docs. |
| hindsight-api-slim/tests/test_recall_dedup.py | New unit tests covering normalization + dedup collapse behavior across thresholds and fact types. |
| hindsight-api-slim/tests/test_recall_config.py | Extends config tests to include recall_dedup_threshold defaults, env var constant, and env parsing validation. |
| hindsight-api-slim/hindsight_api/engine/search/dedup.py | New implementation of recall raw-fact deduplication helpers. |
| hindsight-api-slim/hindsight_api/engine/memory_engine.py | Reads recall_dedup_threshold from resolved config and applies dedup after observation preference and before truncation/token filtering. |
| hindsight-api-slim/hindsight_api/config.py | Adds env var constant, default, config field, validation, and env parsing for recall dedup threshold. |
| .env.example | Documents the new recall dedup env var in the repo root env template. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+60
to
+64
| is_duplicate = any( | ||
| retained_text == normalized_text | ||
| or SequenceMatcher(None, retained_text, normalized_text).ratio() >= threshold | ||
| for retained_text in retained_texts | ||
| ) |
Comment on lines
4149
to
4152
| budget_config_dict = await self._config_resolver.get_bank_config(bank_id, request_context) | ||
| thinking_budget = _resolve_thinking_budget(budget_config_dict, budget, max_tokens) | ||
| recall_dedup_threshold = float(budget_config_dict.get("recall_dedup_threshold", DEFAULT_RECALL_DEDUP_THRESHOLD)) | ||
|
|
Comment on lines
+4947
to
+4953
| if recall_dedup_threshold > 0.0 and scored_results: | ||
| dedup_start = time.time() | ||
| from .search.dedup import collapse_near_duplicate_raw_facts | ||
|
|
||
| before_dedup = len(scored_results) | ||
| scored_results = collapse_near_duplicate_raw_facts(scored_results, recall_dedup_threshold) | ||
| dropped_dedup = before_dedup - len(scored_results) |
Cybercommanders
force-pushed
the
fix/recall-dedup-threshold
branch
from
a8111fd to
e23db8a
Compare
Author
|
Closing at maintainer/requester direction. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification